• 专利附录
  • 专利说明
  • 权利要求
  • 类似技术
  • 同族专利
  • 引用文献
  • 法律状态
  • 优先权
    • 专利摘要:
    Improved method and system for the determination of fruit characteristics using hyperspectral images. The present invention proposes a method and system for measuring the characteristics of fruits before harvest using hyperspectral imaging technology. This procedure includes the optimal capture of images of the fruit on the plant (tree, vine, bush ...) in the field (under ambient conditions), its processing, obtaining analytical results and presenting them to the user; it also allows the creation of images where quantitative analysis results are displayed on color maps. (Machine-translation by Google Translate, not legally binding)
    公开号:ES2795499A1
    申请号:ES201930443
    申请日:2019-05-21
    公开日:2020-11-23
    发明作者:Pulido Francisco José Rodríguez；Muñoz Cristina Gutiérrez；Martín Daniel Talaván；Caballero Raúl Felipe Guzmán
    申请人:Ambling Ingenieria Y Servicios S L；
    IPC主号:
    专利说明:

    [0002] Improved method and system for the determination of fruit characteristics using hyperspectral imaging
    [0004] TECHNICAL FIELD OF THE INVENTION
    [0006] The present invention refers to the field of monitoring agricultural products and more specifically, to an improved method and system for obtaining certain characteristics of fruits before harvest (which allow determining, for example, both the evolution of the state of the fruit maturation as well as its quality) by capturing, processing and analyzing hyperspectral images.
    [0008] BACKGROUND OF THE INVENTION
    [0010] In the field of agriculture, it is often desirable to analyze agricultural products and especially fruits to determine one or more characteristics of interest. For example, farmers who harvest fruits often use some physical and / or chemical characteristics of the fruit to estimate its quality, maturity, appearance or needs for fertilizers or chemical products, among others.
    [0012] For this reason, the knowledge of the physicochemical state of the products of the agri-food industry has always been a fundamental aspect in agriculture, although it is even more so in recent times. First of all, it is necessary to monitor the ripening of the fruit to help decide the optimal time for harvesting. Historically, this process has been carried out in a sensory way by farmers who, based on their experience, estimate the point at which a fruit should be picked. However, intensive agriculture and globalization have made this task practically unapproachable, which is why fast, precise and independent methods of the farmer's experience are required to carry out these measurements on fruits.
    [0014] In today's agricultural industry, the importance of planning is key, being essential a proper decision-making during cultivation and harvesting, to achieve a product of the highest quality with the highest yield and optimal logistical and organizational management between producer, cooperative and processing industry. This is especially important in some fruits such as tomatoes; since the tomato plant does not present fruit maturation in a uniform way, but is generally staggered over a period of time, so that an early harvest will lead to a high% of green or immature fruits, which will be discarded in the harvester machine. In addition, at the factory entrance, the tomatoes delivered by the farmer will be sampled, penalizing green fruits. If, on the other hand, harvesting is delayed, there will be a higher% of overripe fruits, which either remain on the ground and are not collected by the combine, and if they go to the factory they will be equally penalized. Taking into account that both green and overripe fruits are penalized, that is to say, that the farmer will see their benefits diminished, the optimal time of harvest becomes a critical process to optimize. In addition, productions are increasingly oriented to satisfy the demand for quality products, prices being increasingly linked to this factor. On the other hand, and due to the increase in legal requirements in matters of food safety, awareness about healthy eating and high competition, rapid and automated analysis of analyzes are necessary, not only of maturity but also of quality in general. , that allow to evaluate a high proportion of the total production and what is more important, that this analysis is non-destructive.
    [0016] Methods that have traditionally been used to evaluate fruit quality include both sensory analysis (eg visual inspection or tasting) and analytical laboratory techniques. Sensory methods tend to be time consuming and potentially subjective. Analytical laboratory techniques have great disadvantages such as being long and tedious procedures, destruction of the sample or the need for specialized personnel. Samples must be collected, bagged, labeled, dried, and finally sent to the laboratory, ground, and analyzed for component analysis. This excessive handling of the sample adds both time and cost to the analysis. Farmers would prefer to make informed decisions regarding fruit before harvest and these types of methods do not lend themselves to evaluation in the field, so they are interested in accessing analytical techniques that can quickly and efficiently evaluate the qualities of the fruit.
    [0018] Optical techniques have also been used for the analysis of the physicochemical state of the fruits. Thanks to advances in electronics and specifically, to the improvement of radiometric sensors, it is possible to measure the interaction between electromagnetic radiation and matter in a fast and precise way. Once that interaction is measured, it is not Very powerful computation is required to process spectral information and obtain the result of an analysis. Within optical techniques, hyperspectral imaging (HSI) techniques have revolutionized the way of evaluating quality in the agri-food sector, among other reasons because they involve fast, non-destructive procedures, free of chemical reagents. A hyperspectral or hypercube image is a battery of images of the same scene where each one represents the reflectance at a single wavelength, so that we can know the spectral profile of any of the points that make up said image.
    [0020] There are currently some commercial portable equipment that performs spectroscopic measurements such as Viavi MicroNIR Pro®, Konica Minolta CM-700d®, Felix Instruments F-750® or WiSci: Wireless Spectrometer® that differ mainly in the useful spectral range. Among them, the Konica Minolta CM-700d® equipment stands out (which, being specifically a hand-held spectrocolorimeter, provides colorimetric information in terms of the CIELAB color space) or the Viavi MicroNIR Pro® (which, in addition to acquiring reflectance spectra in the near infrared range, it is capable of carrying out spectroscopic transformations such as conversion to absorbance units, carrying out transformations as standard normal variate or derivation of spectra). However, none of these equipment is capable of measuring several fruits at the same time, since they measure by direct contact with the sample; Furthermore, measurements are made in the visible range of the spectrum and from RGB cameras ("Red, Green, Blue", red, green, blue), so that, although there are many solutions based on artificial vision, if the characteristic to be determined cannot be identified in the visible spectrum, these techniques are not valid.
    [0022] The use of hyperspectral cameras is not yet so standardized and, although there are more and more cameras that allow use in real environments, there are no optimal comprehensive solutions for analysis in the agri-food sector.
    [0024] There are some technical solutions for hyperspectral analysis to obtain fruit characteristics, but they are carried out in the laboratory (that is, a fruit or a sample of the fruit of the tree must be collected, transferred to the laboratory, and its characteristics analyzed using hyperspectral techniques). Therefore, they present the disadvantages that we have indicated above for laboratory analytical techniques (they consume time, they are not done in real time automatically, they involve manipulation of the fruits, they are done on a collected sample and not on the total of the fruits to be harvested ...). Also exist solutions that capture hyperspectral images of the field through the use of drones, but due to the distance between the samples and the camera, they do not serve to discriminate and obtain characteristics of the individual fruits, but rather work on the entire crop to obtain macroscopic properties of the crop, such as irrigation, fertilization, number of plants or trees ...
    [0026] In view of the problems of existing solutions, there is a need for a procedure that allows monitoring the quality and ripening of the fruit during the growing cycle, so that they can be known in a fast, reliable, non-destructive, objective, clean way. , versatile, without manipulation of the fruit and in real time of the parameters of quality of the fruit, as well as the optimum moment for the harvest, allowing a better and faster planning and management of the plantation, also avoiding yield losses.
    [0028] SUMMARY OF THE INVENTION
    [0030] The present invention proposes an improved method and system for the determination (measurement) of characteristics and / or properties in products of the agri-food industry and more specifically in fruits (especially tomatoes and grapes but can also be applied to any other fruits) from hyperspectral imaging technology. This procedure includes the methodology for the optimal capture of images of the plant (tree, bush, video) where the fruit grows, the processing of hyperspectral images and the obtaining of analytical results; it also allows the creation of images where quantitative analysis results are shown on color maps.
    [0032] Obtaining the characteristics (physical or chemical parameters) will be carried out from the generation of spectral signatures that allow detecting changes in the spectral composition and detection of substances of interest, including wavelengths outside the visible range of the spectrum, extracting and defining those spectral signatures (patterns) that are related to the different parameters. From a training process using machine learning, the relationships that exist between the spectral profile and different physicochemical properties of the fruit will be established. These properties may be qualitative, such as detection of a certain parameter or the presence of undesirable compounds (pesticides, e.g. gas) or quantitative, such as the concentration or degree of intensity of a compound or parameter (degree of maturation, sugar concentration, acidity, firm.).
    [0033] From the spectral signatures (patterns) defined for each parameter, the different characteristics of a fruit by capturing, processing and analyzing spectral images of said fruit before harvesting it (that is, on the tree or plant). Monitoring (determination / estimation of characteristics) through the use of hyperspectral images allows the quantification and characterization of the parameters objectively, accurately and quickly without the need to manipulate the plant material.
    [0035] In one embodiment of the present invention the most common spectral range is used in this type of instrumentation. This covers the visible range from 400 nanometers and reaches the near infrared up to 1000 nm. This interval is very useful, since it is capable of detecting changes in compounds that affect the visible region, but at the same time it is capable of considering other changes that do not have to affect the appearance of the fruits. This is just an example and in general any range of wavelengths can be used, preferably covering part of the non-visible spectrum.
    [0037] It should be noted that the capture of the images of the fruit will be done without collecting the fruit, that is, it will be done on the tree, vine, bush (in general, plant) where the fruit grows. In addition, the measurement will be carried out only on the fruits, discriminating them from the rest of the vegetation and elements of the image (through an object identification process called segmentation in image analysis). Thanks to segmentation, there is the possibility of independently identifying and characterizing both fruit and other elements (leaves, branches, soil ...).
    [0039] The system can also allow the presentation of the results obtained in an electronic device of the user (mobile phone, PC, tablet, laptop or any other type of electronic device) and can allow the historical storage of the parameters for each fruit of a plot for know the status and evolution of the harvest (for example, quantifying production and quality in different areas of the same plot).
    [0041] In addition, the procedure will be carried out automatically. In other words, the proposed system will capture the images of the fruit, process them, determine from them the different parameters of interest in the fruit and store the results and / or present them to the user (for example, the farmer) on their electronic device. . Therefore, the entire procedure is carried out, without requiring specific user training or knowledge of the technology that is applied.
    [0042] Thus, the proposed technique will allow to carry out the control (of quality, maturity, safety ...) of the fruit in a non-destructive way during the growing cycle (before harvest), automatically, easily, in real time and improving the accuracy, quality, speed, safety, reliability and costs in the processes, allowing a better planning of the crop and harvest.
    [0044] In a first aspect, the present invention proposes a method for determining a group of fruit characteristics, where the method comprises the following steps:
    [0046] - Capture, with natural light, at least one hyperspectral image of at least one part of a fruit plant with a camera (hyperspectral) in the field where the plant is planted (in the field), that is, the camera will be located in the land where the plant is planted; and where in said hyperspectral image, reflectance data of the image pixels are obtained at different wavelengths of a range of the (frequency) spectrum;
    [0048] where the method also comprises the following steps performed by an electronic processing equipment:
    [0050] - Receive the hyperspectral image data and from them, automatically select those pixels belonging to the image of at least one fruit, applying a classification algorithm;
    [0052] - For each of the characteristics to be determined, compare the data obtained from the hyperspectral image for each of the selected pixels for each wavelength of the range, with a pre-established spectral signature (obtained in a previous training process) that corresponds to said characteristic of the at least one fruit;
    [0054] - From said comparison of the previous step, determine the amount of each of the characteristics of the group of at least one fruit in each of the selected pixels, from the comparison of the previous step.
    [0056] In one embodiment, the group characteristics are internal characteristics of the at least one fruit. In one embodiment, the group of characteristics can comprise one or more of the following: soluble solids, pH, firmness, lycopene content, BRIX grades, polyphenols, the external appearance of the at least one fruit or any other characteristic thereof (In general , it can be said that any physical parameter can be obtained by this procedure or chemical of the fruit and especially those that determine its quality or degree of maturity).
    [0058] In one embodiment, the method further comprises: presenting the electronic processing equipment to a user (through a user interface), the quantity determined for each pixel of each of the characteristics and / or the electronic equipment sending, through a communication network , to an electronic device of the user the quantity determined for each pixel of each of the characteristics.
    [0060] In one embodiment, the method also comprises making a color map from the results obtained and, for this, the following steps can be carried out: - carry out a reconstruction in the visible spectrum, of the image of at least one fruit; - for each characteristic, representing each pixel of the image in a color depending on the quantity of said characteristic determined for each pixel.
    [0062] In one embodiment, the distance at which the camera is placed from the plant depends on at least one or more of the following parameters: the size of at least one fruit, the spatial resolution of the camera, the optical technology of the camera, and lighting (in general, preferably the distance is calculated according to one or more of these parameters, so that the fruit to be analyzed has a desired minimum size in the image).
    [0064] In one embodiment, the image is captured when natural light is incident at approximately 45 ° from vertical.
    [0066] Preferably, the spectrum range in which the camera collects hyperspectral image data is a wavelength range between 400 and 1000 nm.
    [0068] In one embodiment, prior to image capture, a camera calibration is performed using a valid element for calibration (for example a calibration frame) placed next to the object to be captured (at least a part of the plant).
    [0069] In one embodiment, the selection of the pixels is done by automatic learning methods (support vector machine).
    [0071] The electronic equipment and the camera can be part of the same device or be separated and communicated through a communication network and in this second case, the electronic equipment receives the hyperspectral image captured through the communication network (or it is downloaded to the electronic equipment after capturing it).
    [0073] In one embodiment, the camera is installed on a vehicle that moves across the land where the plant is planted.
    [0075] In one embodiment, the spectral signature of each characteristic comprises a vector with the reflectance value, for each wavelength of the range, obtained during a previous training process, which corresponds to the presence of a quantity (for example, a unit) of this characteristic in a fruit.
    [0077] In a second aspect, the present invention proposes a system for determining a group of fruit characteristics, where the system comprises the following steps:
    [0079] - A camera to capture, with natural light, at least one hyperspectral image of at least a part of a plant with fruits, where the camera to capture the image is located in the field where the plant is planted and where said hyperspectral image includes data reflectance of the pixels of the image at different wavelengths of a range of the spectrum;
    [0081] - At least one electronic processing equipment configured to:
    [0083] - Receive the hyperspectral image data and from them, automatically select those pixels belonging to the image of at least one fruit, applying a classification algorithm;
    [0085] - For each of the characteristics of the group, compare the data obtained from the hyperspectral image for each of the selected pixels for each wavelength of the range, with a pre-established spectral signature that defines said characteristic of the fruit;
    [0086] - Determine the amount of each of the characteristics of the group of at least one fruit in each of the selected pixels, based on said comparison.
    [0088] Finally, a computer program is presented that comprises computer-executable instructions to implement the described method, when executed on a computer, a digital signal processor, an application-specific integrated circuit, a microprocessor, a microcontroller or any other way. programmable hardware.
    [0089] Said instructions may be stored on a digital data storage medium.
    [0091] For a more complete understanding of these and other aspects of the invention, its objects and advantages, reference may be made to the following specification and the accompanying figures.
    [0093] DESCRIPTION OF THE FIGURES
    [0095] To complement the description that is being made and in order to help a better understanding of the characteristics of the invention, according to some preferred examples of practical embodiments thereof, a set of drawings is attached as an integral part of this description. where, for illustrative and non-limiting purposes, the following has been represented:
    [0097] Figure 1 schematically shows the system in operation according to an embodiment of the invention.
    [0099] DETAILED DESCRIPTION OF THE INVENTION
    [0101] The present invention proposes an improved and comprehensive method and system for obtaining various characteristics of fruits from hyperspectral images.
    [0103] The proposed method and system allows, among other things, to acquire and process the images in the field (under ambient conditions) without collecting the fruit (or fruits) or manipulating it or transferring it to a laboratory for analysis; automatically identify (discriminate) the fruit within the image; obtain various parameters of the fruit (multiparametric characterization); estimate the amount of different physicochemical parameters of interest in fruit; represent the results on color maps based on the identified fruits and the parameters obtained for each fruit; present the information to the user on their electronic device (with the possibility of creating automatic results reports).
    [0105] These characteristics obtained will be internal characteristics of the fruit such as (this is a non-limiting list) soluble solids, pH, firmness, lycopene content, BRIX grades, polyphenols; optionally you can also obtain some external characteristic of the fruit (related to the external appearance) such as color, spots, roughness ... In general, it can be said that any physical or chemical parameter of the fruit can be obtained through this procedure (and especially those that determine its quality or degree of maturity ).
    [0107] In one embodiment, the acquired hyperspectral images, regardless of their spatial dimensions, will preferably have a spectral range of 400 to 1000 nm. On the other hand, although in some examples the tomato is mentioned as a fruit, the invention is proposed and can be extended to any other type of fruit.
    [0109] In a preferred embodiment, the monitored fruits will be tomatoes although the present invention can also be applied to grapes, olives or any other type of fruit.
    [0111] In figure 1 the different parts of the system can be seen schematically. First we have the camera (1) to capture the hyperspectral images of the fruits (2) on the tree. Such a camera (called a hyperspectral camera or hyperspectral sensor) can be of any known type. In one embodiment, said camera will obtain images of axb pixels (also called spatial resolution of the camera, where a is the number of rows of pixels and b the number of columns of pixels, for example, 512x512 pixels although any other number of pixels is possible) in the range from 400 to 1000 nm with a spectral resolution of approximately 3 nm (therefore, in this example, reflectance measurements will be obtained in the camera for 204 different wavelengths, or in other words, for 204 bands of different lengths cool).
    [0113] Then there will be an electronic processing equipment (4) (also called simply a processing unit, equipment, module or device or more simply processor) that receives the captured images and processes them to obtain the desired parameters (although for simplicity, we are talking about an equipment There may be a single unit that performs all the processing or there may be several units, each of which performs some of the processing actions). The processing equipment can display the results to the user (through a user interface such as a screen) and / or it can send the results obtained to an electronic device of the user (for example, a mobile phone, a computer, etc.) Through a wireless communication network, mobile telephony, wired network or of any kind. The user's electronic device and / or the processing equipment can store the numerical results (values of the parameters obtained) and even the possible color maps generated, in files that can be later mass processed.
    [0115] In figure 1 said processing equipment is a laptop but this is only a non-limiting example, any type of electronic data processing equipment can be used.
    [0117] The electronic processing equipment can be connected to the camera through any type of communication network (wired or wireless, mobile phone network, etc.) using any type of technology, such as Bluetooth, mobile phone or any other. Also, in one embodiment, a permanent connection between the camera and the computer equipment is not necessary. The camera may have a storage unit for the images, which will allow their subsequent processing by transferring (downloading) the information to the corresponding electronic processing equipment (as shown in the figure on the right in figure 1).
    [0119] On the other hand, the camera and the electronic processing equipment can be separated as in figure 1, but they can also be within the same housing as part of the same equipment (that is, the same equipment captures the images and processes them to obtain the results). Or there are various processing equipment, some of which are incorporated in the camera and others separate from it (or in other words, the camera can perform, in addition to capturing, some of the actions for processing images).
    [0121] In the proposed procedure, two main stages are distinguished, acquisition (capture) of the image (carried out in the camera) and processing of the image and obtaining results (carried out in one or more processing equipment). The results obtained can then be processed in different ways to present them to the user. Each of these stages will have different steps or actions.
    [0123] Image acquisition (this is just an example of the steps to be performed and not all steps are essential and therefore necessary for the proposed procedure):
    [0125] - Position the camera (1) to capture the image. Preferably, the camera will point perpendicular to the object (2) whose image you want to capture (part of the tree or in general, of the plant where the fruit or fruits to be monitored are), although in other embodiments the orientation may be different. In one embodiment, for greater stability of the captured photo, the camera will be arranged on a support or tripod (3).
    [0127] Although in figure 1, the camera appears in isolation and must be transported from one tree to another to take the photos, in an alternative embodiment the camera may be installed in a device with movement capacity (for example, a tractor, harvester or any type of mobile machine or vehicle) that moves through the crop field, stopping and taking different photos of each tree whose fruits are to be analyzed.
    [0129] Whether the camera is isolated or mounted on some type of machine (for example, a vehicle), for the acquisition of the image it must be placed in a way that allows a direct and close view of the fruit (or fruits) that allows the capture of an image . It must also be placed at a distance compatible with the working distance of the optics used by the camera. The spatial resolution of the camera sensor, together with the specific characteristics of the optics, entails the use of specific working distances to obtain a sufficient definition, which will be adjusted to the characteristics of the sample to be analyzed (fruit). For example, the relationship between the distance at which the camera is placed from the fruit to be analyzed can be given according to the relationship L ( m) = 1,024, where R (px / mm)
    [0130] R ( px /)
    [0131] is the resolution in pixels per millimeter and L is the distance measured between the fruit and the camera lens (this is just an example, and other formulas can be used to obtain the relationship between resolution and distance); this distance must be within the range of distances in which the optics are able to focus correctly. The ratio 1,024 is calculated specifically for the camera used in the present example (where the spatial resolution is 512 x 512 pixels).
    [0133] In general, it can be said that the distance between the camera and the fruits will depend on factors such as the size of the fruit, the spatial resolution of the camera sensor, the optics used and the lighting. The identification of the fruits is based on a spectral discrimination with respect to the rest of the elements of the image, but it is convenient that the fruit covers an area of pixels that allows, on the one hand, to recognize the morphology of the fruit and, on the other hand, to perform a statistic from a specified number of pixels from its mean spectrum. For example, in a preferred embodiment, a distance that makes the fruit or fruits have a minimum size of 500 pixels in the image.
    [0135] In hyperspectral imaging, lighting control is critical since the reflectance spectrum obtained is a relative measure that depends on the incident radiation on the sample; For this reason, the vast majority of works with this type of images are acquired in the laboratory, where it is easy to control the light. Thus, in the solutions proposed in the state of the art, images are captured in a laboratory on samples already harvested (and therefore, laboratory equipment with controlled lighting is used). In the present invention, however, the images are captured in the field where the fruit bearing plant is planted, and where natural light is therefore used (which can vary in an uncontrolled manner). Taking these aspects into account, the proposed methodology preferably includes a system that calibrates the lighting for each of the captures, which makes it robust to changes in environmental conditions. This calibration can be done by measuring the reflectance spectrum of a reference material; the spectrum of this material is stable over time, so the spectral profile of the samples is measured relative to this material, eliminating dependence on environmental conditions. In addition, in the computational part (in the processing equipment or in the camera itself) there are spectral preprocessing tools that minimize the possible spectral inconsistencies caused by irregular lighting. Thus, the extrapolation of laboratory image processing models (with controlled lighting) to images captured in natural light is not trivial.
    [0137] In the present invention, the illumination is direct from the sun. In this case, in a preferred embodiment, the image will be taken if possible when the light (from the sun) falls at a 45 ° inclination with respect to the vertical, approximately. In any case, if the image cannot be taken at 45 ° with respect to the vertical, it should be avoided to take it at the times when the light is more zenith since the quality of the photograph worsens. This geometry is the most widely used since it is the one that offers the best results in spectral reflectance measurements.
    [0139] - Once the camera has been placed (at the appropriate orientation, distance and lighting conditions), the image calibration is preferably carried out. For this, for example, a valid element for the calibration (such as a calibration frame) is placed facing the camera and next to the object whose image is to be captured and the camera calibration is performed (via the corresponding option on the camera or later during image processing). The calibration frame consists of a geometric figure (for example, in the form of a square frame), with an area wide enough to be represented in the image by a set of pixels sufficient for the execution of the normalization, which serves as a reference for white as it is composed of a material that has a maximum reflectance throughout the spectral range, so it will preferably be made of PTFE (polytetrafluoroethylene).
    [0141] - Finally the desired image or images are captured.
    [0143] Image processing and obtaining results (this is just an example of the steps to be carried out and not all the steps are essential and, therefore, necessary for the proposed procedure):
    [0145] - As we have indicated, although we speak of a hyperspectral "image", in truth, what is obtained is a battery of images of the same scene (of the same object) where each one represents the reflectance at a wavelength of the spectrum. So in the end, what you get from the hyperspectral camera is a three-dimensional matrix (called a hypercube) with the reflectance of each point in the image (pixel) at each wavelength.
    [0147] For its processing, the first thing that is done is the conversion of said three-dimensional matrix or hypercube to a flat matrix (two-dimensional). This process is called unfolding or better known by its term in English, “unfolding”. In the example cited above where the images are axb pixels and data is captured at 204 different wavelengths, the dimensions of the matrix go from being axbx204 (three-dimensional) to bx204 (two-dimensional).
    [0149] - Now we carry out the identification of the pixels that correspond to the region of interest (fruit or fruits whose characteristics are to be obtained). For this, in each of the individual spectra (of each pixel), it is evaluated by means of a classifier whether or not they belong to the fruit. This classification is multivariate and the 204 wavelengths available for each point in the image have a different influence. This classifier can use any known tool (for example, in support vector machine, from English "Support Vector Machine, in artificial intelligence), normally based on a (previous) training where the characteristics of the pixels that belong to the fruit are defined (which will vary for each type of fruit). The result is a vector of length equal to the number of spectra (which is coincident with the number of pixels), where for each pixel the membership of the region of interest is indicated with 0 and 1. The sum of this vector is the number of pixels (F) pixels where there is a sample (fruit) in the image. In one embodiment, the classifier used to identify the pixels that belong to the region of interest is based on searching for hyperplanes that separate the samples within the 204 available dimensions. The classification is carried out from some images used for the training of the classification algorithm which, once set-up, is verified by checking its suitability with external samples. During the fine-tuning of the proposed solution, the classification models used are preferably verified with dozens of images available, to ensure that they are as accurate as possible.
    [0151] - Then the vector is folded to the spatial dimensions of the hypercube. The result is a flat matrix (axb) where these regions are shown in the hypercube.
    [0153] - Next, the spectra of the selected pixels are collected, that is, the pixels that belong to the region of interest (to a fruit) that have been assigned a 1 in the classification carried out previously. The (x, y) coordinates of these points (pixels) in the image are also collected. The result is a matrix (Fxnumber of wavelength bands, 204 in the example used above) with the spectra of interest and another matrix (Fx2) with the coordinates, where F is the number of pixels of the image that has been considered that belong to a fruit
    [0155] In this step, from the spectra obtained in the previous step and from the spectral signature previously obtained for each parameter and each type of fruit, the measurement of one or more parameters of interest in the fruit being processed is determined (estimated). monitoring. For this, in a previous training process (using for example artificial intelligence techniques such as "machine leaming") the spectral signature (also called spectral pattern ) has been obtained that characterizes each of the parameters to be measured in each type of device. fruit. That is, test spectral images of different fruits have been captured in which the measurement of the parameters of interest was known, the data of these hyperspectral images has been correlated with the amount of each parameter present in each fruit and, from there, the value has been obtained at each of the wavelengths that quantifies the presence of each of the parameters. In other words, the spectral signature of each parameter comprises a vector with the reflectance value, for each wavelength that corresponds to the presence of a quantity of said parameter in a fruit of a certain type (grape, tomato ...).
    [0157] To ensure the accuracy of the model, an experimental design has been built for pre-training, which first separates the samples into calibration and validation sets. The models have been trained with the calibration samples and the suitability of the models has been verified with the validation samples. There are statistical tools and quality parameters in chemometrics that ensure their correct operation, the lack of unwanted overfitting phenomena and that at the same time ensure that the model will continue to function with new samples in the future.
    [0159] In one embodiment, in the case that the analysis is quantitative, what is obtained is a multivariate regression model such as, for example, partial least squares regression (PLSR) or linear multiple regression (MLR). The final result of these training models is a vector with a coefficient for each wavelength plus an independent term that, applied to a problem spectrum, yields a numerical value that corresponds to the magnitude to be analyzed. These coefficients depend on the analyte (that is, on the physical or chemical parameter to be measured) and on the specific fruit (tomato, grape, olive, apple).
    [0161] Thus, in the training process, a vector (pattern or spectral signature) will be obtained for each analyte and each type of fruit to be monitored. In the specific case mentioned above, these vectors will have 204 coefficients (one for each of the 204 wavelengths of the hyperspectral image) plus an independent term. Next, the spectrum of the captured image is compared with these coefficients of the spectral signature to estimate the presence of said analyte (characterized by the spectral signature) in the fruit or fruits whose image has been captured. To do this, in one embodiment, for each of the F pixels resulting from the hyperspectral image captured, each value of the spectrum of the pixel (for each wavelength) obtained from the hyperspectral image (s) is multiplied. ) captured (s) (previous step) by its corresponding coefficient, add the values obtained and add the independent term. The data obtained is the result of said analyte at that point (pixel). This procedure is applied to each of the F spectra and the results matrix is obtained with the estimated measurement of the parameter under study (analyte) in each pixel of the fruit (this is just an example and the comparison to obtain the results can be performed in any other known way).
    [0163] Processing and presentation of results
    [0165] - The results obtained in the previous step can be processed in different ways to present them to the user. The data obtained can be exported to a file (for example, an xlsx file) showing the raw data or with some type of statistical analysis (mean value of each of the analytes in the image, dispersion of the results, etc.). This file can be stored on the electronic processing equipment and / or sent to an electronic device of the user for him to access (and store it on his device if he so wishes).
    [0167] - Also so that the results are more easily understood and visualized by the user, they can be presented in the form of a color map of the identified fruits. In these maps, through a color grading for each pixel, a numerical scale is visually represented so that, at a simple glance, it is possible to see the magnitude of a parameter within an image. Thus, the results can be accompanied by the resulting color map to improve the understanding of the results.
    [0169] Although many embodiments of the present invention have referred to tomatoes and grapes and to the measurement (estimation) of certain parameters, the present invention is applicable to any other type of fruit and parameters.
    [0171] In this text, the term "comprises" and its derivations (such as "comprising", etc.) should not be understood in an exclusive sense, that is, these terms should not be interpreted as excluding the possibility that what is described and defined can include more elements, stages, etc.
    [0172] Some preferred embodiments of the invention are described in the dependent claims that follow.
    [0174] Having sufficiently described the nature of the invention, as well as the way in which it is carried out in practice, it should be noted the possibility that its different parts may be manufactured in a variety of materials, sizes and shapes, and those parts may also be included in its constitution or procedure. variations that practice advises, as long as they do not alter the fundamental principle of the present invention.
    [0176] The description and the drawings simply illustrate the principles of the invention. Therefore, it should be appreciated that those skilled in the art will be able to devise various arrangements which, although not explicitly described or shown herein, represent the principles of the invention and are included within its scope. Furthermore, all the examples described in this document are provided primarily for pedagogical reasons to help the reader understand the principles of the invention and the concepts contributed by the inventor (s) to improve the technique, and should be considered as non-limiting. with respect to such examples and conditions specifically described. In addition, everything set forth in this document related to the principles, aspects and embodiments of the invention, as well as the specific examples thereof, encompass equivalences thereof.
    [0178] Although the present invention has been described with reference to specific embodiments, those skilled in the art should understand that the foregoing and various other changes, omissions, and additions in the form and detail thereof can be made without departing from the scope of the invention such as defined by the following claims.
    [0180] Definition of terms.
    [0182] Some terms used are defined here, for clarity of description:
    [0184] Spatial resolution of the camera sensor: it is the number of pixels that the camera sensor consists of, generally expressed by the product of the pixels of the horizontal line by those of the vertical line in a rectangular sensor.
    [0186] Spectral resolution of the camera sensor: It is the number of wavelengths to which the camera sensor is sensitive within a certain range.
    [0187] Working distance: Range of distances in which the camera's optics are able to correctly focus on the sample.
    权利要求:
    Claims (14)
    [1]
    1. Method for the determination of a group of fruit characteristics, where the method comprises the following steps:
    - Capture, with natural light, at least one hyperspectral image of at least a part of a plant with fruits with a camera in the field where the plant is planted; and where in said hyperspectral image, reflectance data of the image pixels are obtained at different wavelengths of a range of the spectrum;
    where the method also comprises the following steps performed by an electronic processing equipment:
    - Receive the hyperspectral image data and from them, automatically select those pixels belonging to the image of at least one fruit, applying a classification algorithm;
    - For each of the characteristics of the group, compare the data obtained from the hyperspectral image for each of the selected pixels for each wavelength of the range, with a pre-established spectral signature that corresponds to said characteristic of the at least one fruit;
    - From said comparison of the previous step, determine the amount of each of the characteristics of the group of at least one fruit in each of the selected pixels, from the comparison of the previous step.
    [2]
    2. Method according to claim 1 where the characteristics of the group are internal characteristics of the at least one fruit.
    [3]
    3. Method according to claim 1 where the group of characteristics comprises one or more of the following: soluble solids, pH, firmness, lycopene content, BRIX grades, polyphenols or the external appearance of the at least one fruit.
    [4]
    4. Method according to any of the preceding claims, further comprising: - presenting the electronic processing equipment to a user, the amount determined to each pixel of each of the characteristics and / or
    - the electronic equipment will send, through a communication network, to an electronic device of the user the amount determined for each pixel of each of the characteristics.
    [5]
    5. Method according to any of the preceding claims, further comprising:
    - carry out a reconstruction, in the visible spectrum, of the image of at least one fruit;
    - for each characteristic, representing each pixel of the image in a color depending on the quantity of said characteristic determined for each pixel.
    [6]
    6. Method according to any of the preceding claims, where the distance at which the camera is placed from the plant depends on at least one or more of the following parameters: the size of the at least one fruit, the spatial resolution of the camera , camera optics and lighting technology.
    [7]
    7. Method according to any of the preceding claims, wherein the image is captured when natural light falls at approximately 45 ° from the vertical.
    [8]
    8. Method according to any of the preceding claims, wherein the spectrum range in which data from the hyperspectral image is collected is a wavelength range between 400 and 1000 nm.
    [9]
    9. Method according to any of the preceding claims, where before the image is captured a camera calibration is performed using a valid element for calibration placed next to the object to be captured.
    [10]
    10. Method according to any of the preceding claims, wherein the selection of the pixels is carried out by means of automatic learning methods.
    [11]
    11. Method according to any of the previous claims, where the electronic equipment and the camera are part of the same device or are separated and communicated through a communication network and in this second case, the electronic equipment receives the hyperspectral image captured at through the communication network.
    [12]
    12. Method according to any of the preceding claims where the spectral signature Each characteristic comprises a vector with the reflectance value, for each wavelength of the range, obtained during a previous training process, which corresponds to the presence of a unit of said characteristic in a fruit.
    [13]
    13. System for determining a group of fruit characteristics, where the system comprises the following steps:
    - A camera configured to capture, with natural light, at least one hyperspectral image of at least a part of a plant with fruit, where the camera to capture the image is located on the land where the plant is planted and where said hyperspectral image comprises reflectance data of image pixels at different wavelengths over a spectrum range;
    - At least one electronic processing equipment configured to:
    - Receive the hyperspectral image data and from them, automatically select those pixels belonging to the image of at least one fruit, applying a classification algorithm;
    - For each of the characteristics of the group, compare the data obtained from the hyperspectral image for each of the selected pixels for each wavelength of the range, with a pre-established spectral signature that defines said characteristic of the fruit;
    - Determine the amount of each of the characteristics of the group of at least one fruit in each of the selected pixels, based on said comparison.
    [14]
    14. Computer program comprising computer-executable instructions to implement the method according to any of claims 1-12, when executed on a computer, a digital signal processor, an application-specific integrated circuit, a microprocessor, a microcontroller or any other form of programmable hardware.
    类似技术:
    公开号 | 公开日 | 专利标题
    Virlet et al.2016|Field Scanalyzer: An automated robotic field phenotyping platform for detailed crop monitoring
    Saberioon et al.2014|Assessment of rice leaf chlorophyll content using visible bands at different growth stages at both the leaf and canopy scale
    Barbedo2019|Detection of nutrition deficiencies in plants using proximal images and machine learning: A review
    Lee et al.2014|Hyperspectral near-infrared imaging for the detection of physical damages of pear
    McCarthy et al.2010|Applied machine vision of plants: a review with implications for field deployment in automated farming operations
    US20140168412A1|2014-06-19|Methods and systems for automated micro farming
    Narváez et al.2016|LiDAR and thermal images fusion for ground-based 3D characterisation of fruit trees
    JP2006250827A|2006-09-21|Analytical method for growth condition of crop
    Widjaja Putra et al.2018|Enhanced broadband greenness in assessing Chlorophyll a and b, Carotenoid, and Nitrogen in Robusta coffee plantations using a digital camera
    Borhan et al.2017|Evaluation of computer imaging technique for predicting the SPAD readings in potato leaves
    Van De Vijver et al.2020|In-field detection of Alternaria solani in potato crops using hyperspectral imaging
    Chen et al.2019|Applying high-resolution visible-channel aerial imaging of crop canopy to precision irrigation management
    Cubero et al.2018|Application for the estimation of the standard citrus colour index | using image processing in mobile devices
    Gutiérrez et al.2019|On‐the‐go hyperspectral imaging for the in‐field estimation of grape berry soluble solids and anthocyanin concentration
    Bourgeon et al.2016|Field radiometric calibration of a multispectral on-the-go sensor dedicated to the characterization of vineyard foliage
    Matese et al.2013|Mapping of vine vigor by UAV and anthocyanin content by a non-destructive fluorescence technique
    White et al.2019|Determining a robust indirect measurement of leaf area index in California vineyards for validating remote sensing-based retrievals
    Friedman et al.2016|Assessment of leaf color chart observations for estimating maize chlorophyll content by analysis of digital photographs
    Zhou et al.2021|Assessment for crop water stress with infrared thermal imagery in precision agriculture: A review and future prospects for deep learning applications
    Hardin et al.2012|In situ measurement of pecan leaf nitrogen concentration using a chlorophyll meter and vis-near infrared multispectral camera
    Pinto et al.2019|Classification of Hass avocado | in terms of its ripening via hyperspectral images
    Mishra et al.2019|Early detection of drought stress in Arabidopsis thaliana utilsing a portable hyperspectral imaging setup
    Benelli et al.2020|In-field hyperspectral imaging: An overview on the ground-based applications in agriculture
    US20180373937A1|2018-12-27|Methodsand systemsfor automated micro farming
    Awais et al.2021|Assessment of optimal flying height and timing using high-resolution unmanned aerial vehicle images in precision agriculture
    同族专利:
    公开号 | 公开日
    ES2795499B2|2021-12-17|
    引用文献:
    公开号 | 申请日 | 公开日 | 申请人 | 专利标题
    US20180259496A1|2012-06-01|2018-09-13|Agerpoint, Inc.|Systems and methods for monitoring agricultural products|
    US20150027040A1|2013-07-26|2015-01-29|Blue River Technology, Inc.|System and method for individual plant treatment based on neighboring effects|
    WO2016183000A1|2015-05-12|2016-11-17|BioSensing Systems, LLC|Apparatuses and methods for bio-sensing using unmanned aerial vehicles|
    EP3467702A1|2017-10-04|2019-04-10|Kws Saat Se|Method and system for performing data analysis for plant phenotyping|
    法律状态:
    2020-11-23| BA2A| Patent application published|Ref document number: 2795499 Country of ref document: ES Kind code of ref document: A1 Effective date: 20201123 |
    2021-12-17| FG2A| Definitive protection|Ref document number: 2795499 Country of ref document: ES Kind code of ref document: B2 Effective date: 20211217 |
    优先权:
    申请号 | 申请日 | 专利标题
    ES201930443A|ES2795499B2|2019-05-21|2019-05-21|Improved method and system for the determination of fruit characteristics using hyperspectral imaging|ES201930443A| ES2795499B2|2019-05-21|2019-05-21|Improved method and system for the determination of fruit characteristics using hyperspectral imaging|
    [返回顶部]